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ABSTRACT 

This primer is designed to help charter school leaders in 
Ohio in coordinating testing and test data reporting procedures as required 
by state and federal guidelines. The report discusses the roles and 
responsibilities of charter schools under Ohio’s new accountability system 
and what charter schools need to know to administer the tests of the 
accountability system effectively. The responsibilities of charter schools 
regarding testing are outlined, and how charter schools should collect and 
report data is discussed. Charter schools are increasingly being held to the 
accountability standards of traditional district schools and are also subject 
to the requirements regarding student achievement and accountability of the 
No Child Left Behind Act of 2001. Appendixes discuss the Ohio test 
development process and contain lists of the state testing dates and testing 
contacts and information, and a glossary of testing terms. (SLD) 
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INTRODUCTION 



Increasingly, charter schools are being held to the accountability standards of traditional district schools and 
are now also subject to the newest requirements regarding student achievement and accountability under the 
federal No Child Left Behind Act. Unfortunately, few charter schools have the financial resources necessary to 
hire full-time testing coordinators to help them navigate the intricacies of state and federal testing 
requirements. This primer is designed to aid charter school leaders in coordinating testing and test data 
reporting procedures as required by state and federal guidelines. The report seeks to answer the following 
questions: 

1 ) What are the roles and responsibilities of charter schools under Ohio s new accountability system? 

2) What do charter schools need to know to effectively administer the test? 

3) What are the responsibilities of charter schools regarding testing? 

4) How should charter schools collect and report testing data? 

THE ROLES & RESPONSIBILITIES OF CHARTER SCHOOLS UNDER OHIO S NEW ACCOUNTABILITY 
SYSTEM FROM PROFICIENCY TO ACHIEVEMENT & DIAGNOSTIC 

On August 1 5, 2003 Governor Taft signed House Bill 3 (HB3) as a means of aligning Ohio s Student 
Accountability Plan with the requirements of the federal No Child Left Behind Act. Major changes resulting 
from this accountability plan include: 

1 ) the transition from proficiency tests to achievement tests, 

2) the move from four performance designations (advanced, proficient, basic, and below basic) to 
five (advanced, accelerated, proficient, basic, and limited), for both schools and districts, 

3) the requirement that all students be tested, including ESL or LEP students, and 

4) the inclusion of both group and aggregate student results to assess school- and district-level 
performance. 

With this new accountability comes increased challenges, but also multiple and more accurate ways to 
highlight and track the achievement of all students. Charter schools are required to test their students 
following the same requirements that apply to traditional district schools in the state of Ohio. The information 
below outlines the various types of testing required of charter school students in Ohio (achievement testing, 
diagnostic testing, and graduation testing), and then attempts to explain how results are used to determine 
performance designations. Please note that more information about the new accountability plan can be found 
at: www.ode.state.oh.us/Accountabilitv/default.asp 

Achievement Testing 

Previous to HB3, Ohio students in grades 4, 6, and 9 were required to take proficiency tests in five areas 
(Reading, Mathematics, Writing, Science, and Citizenship). Starting this year (2003-4), the state is moving 
toward testing all students in grades 3-8 through achievement tests in Reading, Mathematics, Writing, Science 
and Social Studies, and in grade 1 0 via the Ohio Graduation Test. Readers should note that Ohio s state 
student proficiency testing program, like most standards-based testing programs, relies on criterion- 
referenced tests (CRT) designed to assess students strengths and weaknesses with respect to a specific set of 
standards, skills or knowledge. Typically CRTs result in student classifications such as pass/fail or 
at/below/above standard. 

Norm-referenced tests (NRTs), which include familiar national commercial achievement tests (such as the 
TerraNova and the SAT-9), are used to answer different questions about student achievement and oftentimes 
assess different knowledge and skills than CRTs. For example, a student deemed as below grade level based 
on his/her CRT score may have made substantial progress based on his/her NRT scores. Conversely, a 
students CRT score may classify him/her as above standard but he/she might still be making below expected 
gains according to their NRT scores. As noted earlier, instead of four performance designations (advanced, 
proficient, basic, and below basic), student performance in Ohio will henceforth be reported under five 
designations (advanced, accelerated, proficient, basic, and limited). The move from proficiency testing to 
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achievement testing will not happen overnight, but will instead be phased in over the next five years. That 
phase-in is displayed in the chart on page 3. 

Diagnostic Testing 

Starting in 2003-04, charter schools and traditional district public schools will also be required to administer 
diagnostic tests. Diagnostic tests are used to assess areas of skills and weaknesses in children so that teachers 
can gauge their instructional practices to address these areas. Teachers and schools are required to 
administer diagnostic tests under the following circumstances: 

• Grade 3 writing if a school fails to make Annual Yearly Progress (AYP) for two or more 
consecutive years OR if a school is in School Improvement status under NCLB; 

• Grades 1 -3 if a student transfers into a new school (test must be administered to the child within 
30 days of starting school); and 

• Grades 1 and 2. 

Kindergarten diagnostics are made up of either the diagnostic measure (a longer, more formal test) or the 
screening measure (a shorter version/subset of the diagnostic measure); both of these measures are used to 
test the skills children should have at the conclusion of kindergarten. This assessment differs from the 
kindergarten readiness assessment, being field tested in October 2003. The latter test will measure the skills 
that students bring to the kindergarten classroom; the definition of readiness is based on a level of skills that 
children are expected to have before entering kindergarten. Those skills include reading (e.g., letter 
recognition and sound formation), pre-writing (e.g., letter formation) and mathematics (e.g., counting and 
comparisons). In school year (2004-2005), both elements of the kindergarten diagnostic program will also be 
required: 

• Kindergarten readiness administered within the first six weeks of school, and 

• Kindergarten readiness administered to the kindergarten student who transfers into the building or 
district. 

Similar to the move from proficiency testing to achievement testing, the move toward diagnostic testing will not 
happen overnight but will instead be gradually implemented over the course of the next five years, as set 
forth in the following table. It outlines which test (achievement and/or diagnostic) will be implemented in which 
grades during which years. Those tests in red are diagnostic tests, while those in black font are achievement 
tests. Simultaneous with the implementation of the Achievement and Diagnostic tests will be the eventual 
phasing out of the Ohio Proficiency Tests. The second table depicts the timeline for phasing out the OPT. Note: 
the Grade 9 proficiency tests will be replaced by Ohio Graduation Tests. 
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Schedule for the Implementation of Ohio's Achievement & Diagnostic Tests for K-8 
Black represents Achievement Tests while Red represents diagnostic tests 





2003-04 


2004-05 


2005-06 


2006-07 


2007-08 


Kindergarten 




Readiness 


Readiness 


Readiness 


Readiness 


Grade 1 


Reading, Math, 
Writing 


Reading, Math, 
Writing 


Reading, Math, 
Writing 


Reading, Math, 
Writing 


Reading. Math, 
Writing 


Grade 2 


Reading. Math, 
Writing 


Reading. Math. 
Writing 


Reading, Math, 
Writing 


Reading. Math. 
Writing 


Reading, Math, 
Writing 


Grade 3 


Writing. Reading 


Writing, Reading, 
Math 


Writing, Reading, 
Math 


Writing, Reading, 
Math 


Reading, Math, 

Writing. Science, 
Social Studies 


Grade 4 


NA 


Reading, Writing 


Reading, Math, 
Writing 


Reading, Math, 
Writing 


Reading, Math, 
Writing, Science, 
Social Studies 


Grade 5 


NA 


Reading 


Reading, Math 


Reading, Math, 
Science, Social 
Studies 


Reading, Math, 
Science, Social 
Studies, Writing 


Grade 6 


NA 


NA 


Reading, Math 


Reading, Math 


Reading, Math, 

Writing, Science. 
Social. Studies 


Grade 7 




Math 


Reading, Math 


Reading, Math, 
Writing 


Reading, Math, 
Writing, Science, 
Social Studies 


Grade 8 




Reading, Math 


Reading, Math 


Reading, Math, 
Science 


Reading, Math, 
Science, Social 
Studies, Writing 



Schedule for Phasing Out the Ohio Proficiency Test 





2003-04 


2004-05 


2005-06 


2006-07 


2007-08 


Grade 4 


Reading, Math, 
Writing, Science, 
Social Studies 


Math, Science, 
Social Studies 


OPT is dead 


OPT is dead 


OPT is dead 


Grade 6 


Reading, Math, 
Writing, Science, 
Social Studies 


Reading, Math, 
Writing, Science, 
Social Studies 


OPT is dead 


OPT is dead 


OPT is dead 
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Ohio Graduation Testing 



Under HB3, the class of 2007 will be the first class required to pass the Ohio Graduation Tests to receive a 
diploma. These students will be tested across five subject areas (reading, math, writing, science, and social 
studies) as part of the Ohio Graduation Tests in March 2005 when they are sophomores. The State Board of 
Education did not set passing standards for the March 2003 administration of OGT reading and mathematics. 
The Board has stated its intention to use data from the March 2004 administration to inform the 2005 
standard setting process. Following are the recommendations from the standard setting committees for 
reading and math, and what the impact of those recommended cut scores would have been on students in 
March 2003 if the recommendations had been applied by the State Board: 

Recommended cut scores for OGT reading 51 total points available 

Advanced: 40 out of 51 ; approximately 35% of students would have been in this category. 

Proficient: 28 out of 51; approximately 31% of students would have been in this category. 

Basic: 21 out of 51 ; approximately 1 2% of students would have been in this category. 

Approximately 22% of students would have been in the Below Basic category. 

Recommended cut scores for OGT mathematics 54 total points available 
Advanced: 43 out of 54; approximately 6% of students would have been in this category. 

Proficient: 31 out of 54; approximately 1 8% of students would have been in this category. 

Basic: 21 out of 54; approximately 23% of students would have been in this category. 

Approximately 53% of students would have been in the Below Basic category. 

State law now requires districts and charter schools in academic watch (AW) or academic emergency (AE) to 
administer practice tests to all ninth graders each September. These practice tests must be aligned to the 
Ohio Graduation Tests (OGT). Each district (charter school) shall determine the dates and times of 
administering the tests to its ninth grade students; districts (charter schools) also score the tests. The Department 
is required to make available on its web page OGT practice tests for districts to download and reproduce to 
meet this new requirement. 

OGT practice test materials (for reading and math) are posted at: 
http://www.ode.state.oh.us/proficiency/oqt/oqtmaterials.asp 

Principals of schools with 1 0 1h -graders should have already received a shipment of OGT practice tests for 
2003. These tests are half-length versions of the OGT in mathematics and reading and are intended to prep 
1 0 1h -grade students and teachers on the kinds of Items that will appear on a real test. The practice test is NOT 
a secure test; it may be copied and distributed to students and parents. 

Field Testing: Science and Social Studies (all grades) 

Educators around the state are being asked to volunteer to participate in field-testing and review of social 
studies and science test items. Field-testing will occur in two phases. The first field test will take place October 
20 November 21 , 2003. The second will occur January 3 30, 2004. Teachers who use field-testing 

materials in their classroom will be asked to spend one or more hours of classroom time and to provide 
feedback on the activities completed. Field test participants will receive detailed materials and instructions. 
They will also be able to contact ODE staff by phone or e-mail with any questions. Educators who do not use 
field-testing lessons in the classroom may still review these lessons and provide written feedback and revision 
suggestions. 

You can find more information and the field test registration form by going to the Ohio Department of 
Education home page, www.ode.state.oh.us, and choosing Science and Social Studies Model Curricula Field 
Test Registration from the list of Hot Topics on the right side of the page. The link may also be accessed from 
the science or social studies academic content standards page. The direct link is: 
http://www.ode.state.oh.us/academic content standards/misc/FieldTestParticipationForm.asp 
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Student Intervention 



Results of achievement testing may indicate that students need interventions. Under HB3, student intervention, 
provided by the school and/or district is mandated for: 

- 3rd graders reading below proficient ; 

— Students (any grade) scoring below proficient on achievement tests; 

— Students (any grade) failing to make satisfactory progress toward attaining grade level 

academic standards on diagnostic tests (NOTE: Satisfactory progress has not yet been defined 

by the state.); and 

— 9th graders failing to make satisfactory progress on the OGT practice test. 

Interventions include additional tutoring, individual instruction/smaller pull-out classes, summer school, etc. 
Additional federal or state money may be available for remediation efforts. 

Performance Designations 

Whereas previous designations based on student achievement and performance were assigned only at the 
district level, these same five designation (Excellent, Effective, Continuous Improvement, Academic Watch, and 
Academic Emergency) will now be applied to districts, school buildings, and community/charter schools. 
Additionally, designation will now be determined on multiple measures as opposed to simply a percentage of 
performance (local report card) indicators met. 

Beginning in the 2003-2004 school year, school and district performance designations will be determined 
using all four of the following: 

Percent of performance indicators (Current Ohio Indicators) 

Performance index score 
Improvement (i.e., Growth Calculation) 

Adequate Yearly Progress 

Current Ohio Indicators: Currently there are 22 total indicators based on 20 test scores (5 subject areas and 
4 grades (4 th , 6 th , 9 ,h , and 1 0 th ) as well as graduation rate (goal is 73.6% or improvement over previous 
year), attendance rates (elementary and middle schools only goal is 93.0% or improvement over previous 
year), and whether 75% or more of all students are at or above grade level. However, because of the 
phasing in of achievement tests along with the phasing out of proficiency tests over time, these indicators are 
changing. At this point, all that has been said by ODE representatives is that new Ohio indicators will take 
into account current testing practices (i.e., grades tested and areas) so the total number of indicators may vary 
over time. 

Performance Index: The performance index is designed to be more sensitive to gradations of achievement 
than the current Ohio Indicators by crediting achievement at all performance levels while weighting higher 
performance more than lower performance. Weights for performance designations are indicated below and 
indicate that better performance is weighted more heavily. 



Weights 

Advanced 


1.2 


Below Basic 


0.3 


Proficient 


1.0 


Untested 


0.0 


Basic 


0.6 







Example of how performance index would be calculated fora school: 

Performance Level % of students at level Weight Score 



Advanced 


2.0 


1.2 


2.4 


Proficient 


21.6 


1.0 


21.6 


Basic 


33.3 


0.6 


20.0 


Below Basic 


39.2 


0.3 


1 1.8 


Untested 


3.9 


0.0 


0 


Total 


100 




55.8 


Please be aware that a 


score of 55.8 has no meaning 


alone as beginning this year 


multiple measures 
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will be used to assign ratings to schools and districts. 



Growth Calculation : Growth calculations will be incorporated once grade 3-8 tests are in place and will be 
value-added, meaning that multiple years of data will be used to assess growth over time. A temporary 
measurement has been established based on the following changes in the performance index score: 1 ) two years 
of improvement, 2) at least 1 0 total points of improvement, or 3) at least 3 percentage points of improvement 
in the most recent year. Note: at this point, how much improvement is needed has not yet been stated. 
Based on such improvements, districts or schools can move from Academic Emergency to Academic Watch or 
Academic Watch to Continuous Improvement. 

Annual Yearly Progress : In the past, AYP applied to Title I funded schools and districts only. Under HB3, as 
part of the alignment with No Child Left Behind (NCLB) standards, AYP applies to all public schools and 
districts, regardless of Title I funding. AYP scores will be calculated for all students as a group, as well as 
certain subsets of students determined by students race/ethnicity, disability, LEP status, and poverty. NCLB 
requires districts and states to identify, monitor, and then incrementally intervene in failing schools. If failing 
schools do not begin to make satisfactory progress in student learning, districts (and charter sponsors) are 
supposed to take strong actions to turn them around. 

Adequate Yearly Progress (AYP) measurements will focus on the performance and participation of various 
groups of students, based on race or ethnicity, socioeconomic status, disability and language background. 
Schools need to focus efforts on the improvement of all students, and also in each of these subgroups in order 
to make AYP. Schools and school districts will be required to make AYP in reading and mathematics at all 
grade levels tested. The goal is to have 100 percent of students proficient by the 201 3-2014 school year. 

The AYP provisions in NCLB require that schools are accountable for overall student achievement and the 
achievement of low-income students, students form different racial/ethnic backgrounds, limited -English 
proficient students, and students with disabilities, in the areas of language arts and math. Under NCLB, if a 
school doesn t make AYP for one of these groups, it doesn t make AYP. 

States decide whether schools are making AYP through a five-step process: 

1 ) States determine what all students should know and be able to do. (Note: To find a chart of the 
annual objectives for each grade level test in reading and math, go to 

www.ode.state.oh.us/accountability/defult.asp and click on 2002-2003 Goals for Adequate Yearly 
Progress) 

2) States calculate the starting point for AYP. (Note: Current AYP requirements are shown below) 

3) States set specific targets to measure whether all groups of students make AYP in language arts and 
math. 

4) States measure the performance of students, schools, and school districts. 

5) Steps are taken to help students in schools that do not make AYP. 

Ninety-five percent of all students and all subgroups must participate in the statewide assessment program for 
a school or district to make AYP. The state sets annual objectives for all schools and districts that identify the 
percent of students who must be proficient. Schools and districts, as well as subgroups with 30 or more 
students (45 or more for students with disabilities) must meet these goals to make Adequate Yearly Progress. 

In addition, elementary and middle schools must meet an attendance rate standard (93.0% or improvement 
over last year), and high schools must meet a graduation rate standard (73.6% or improvement over last 
year). As indicated, schools that do not meet the standard can meet AYP if their attendance or graduate rate 
improved from the previous year (although the percent improvement needed has not been specified). 

2003-04 AYP Goals: 

4 th Grade Reading: 40.5% proficient 6 th Grade Reading: 36.0% proficient 

4 th Grade Math: 35.9% 6 th Grade Math: 36.8% 

Note: AYP goals have not been set for 3 rd grade reading tests. 

Schools or districts with subgroups that do not meet the annual objectives for reading or math can meet AYP 
by reducing the proportion of students who are not proficient. If the proportion of students who are not 
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proficient is reduced by at least 1 0 percent from the year before, the school or district meets the AYP 
requirements. This provision is often called safe harbor. For example, if 80% of students in a subgroup were 
deemed not proficient (so 20% were deemed proficient), then next year the percent not proficient must be 
reduced by 1 0% of 80% (or 8%) to 72% (thus resulting in 28% of students in the subgroup rated proficient). 

If a school does not make AYP for the previous school year, it is identified as in need of improvement. Under 
the law, there are no consequences for not making AYP for a single year. If a school fails to make AYP for 
two consecutive years, however, parents need to be given the option to transfer their children to a higher 
performing school in the district (or a charter school). If a school fails to make AYP for a third consecutive 
year and is a Title I school, families have the right to take a portion of their federal dollars and purchase 
tutoring and other supplemental education services from sources other than their schools. 

If a school does not make AYP for four years, it is identified for corrective action. Children can continue to 
transfer to other schools or receive tutoring and other services. In addition, the district (or charter sponsor) is 
expected to take at least one of the following corrective actions: 

• Replace the school staff. 

• Institute a new curriculum. 

• Significantly decrease management authority at the school level. 

• Appoint an outside expert to advise the school. 

• Extend the school year or school day. 

• Restructure the school. 

If the school fails to make AYP for five years, the school system must develop an alternate governance 
plan for it, including at least one of the following: 

• Reopen the school as a public charter school. 

• Replace all or most of the staff responsible for the lack of progress. 

• Enter into a contract with a private company to operate the school. 

• Turn over operation and management of the school to the state. 

• Implement other fundamental reforms approved by the state. 

If a school does not make AYP for six years, the alternate governance plan that was developed the 
previous year must be implemented. 

Combining the Measures 



Shown below is ODE s plan for how the various measures will be used to calculate school/district ratings for 
the 2003-04 academic year. Note that a growth measure is not included this year as achievement tests are 
being phased in beginning this year and thus no pre-score is available to measure gain over time. 



Rating 


Indicators Met 




Performance 
Index Score 




AYP Status 


Excellent 


94-100% 


or 


100-120 


and 


Met/Not Met 


Effective 


75-93.9% 


or 


90-99.9 


and 


Met/Not Met 


Continuous 

Improvement 


0-74.9% 


or 


0-89.9 


and 


Met 


50-74.9% 


or 


80-89.9 


and 


Not Met 


Academic 

Watch 


31-49.9% 


or 


70-79.9 


and 


Not Met 


Academic 

Emergency 


0-39.9% 


and 


0-69.9 


and 


Not Met 



WHAT CHARTER SCHOOLS NEED TO KNOW TO EFFECTIVELY ADMINISTER TESTS 
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Charter schools need to recognize that all students are required to be tested regardless of IEP or LEP 
status. Students with documented disabilities may take an alternate assessment, subject to availability, as 
these tests are currently being developed. More information on alternate assessments is available at the Ohio 
department of education webpage: 

http:/ /www.ode.state.oh.us /proficiency /Alternate Assessment/default.asp 

Charter school principals and staff also should know that practice tests (proficiency tests, 3 rd . grade reading 
achievement tests, and Ohio Graduation tests) are available for their use and can be ordered or 
downloaded from the web. Additional information and practice tests are available at 
http:/ /www.ode.state.oh.us /prof iciency/samples.asp 

In addition, the following link: http://www.ode.state.oh.us/proficiency/Diaqnostic Achievement/Default.asp 
routes to a page where Ohio State Academic Content Standards can be found. 

Charter school administrators and staff need to realize that Ohio s state mandated tests are designed to 
assess student s knowledge of the state mandated academic content standards. The best way to prepare 
students to achieve at their highest potential is to make sure they understand the material they will be tested 
on and what the test will consist of. Aligning instruction with curricular standards and providing students 
examples of past tests are the two greatest things that charter school staff can do to help their students 
succeed. 

Last, charter schools are required to track students scores overtime and report the test results (i.e. fall and 
spring administration or winter and spring administration) as requested by the state. NOTE: More 
information on testing dates and fall/spring administrations of tests is given beginning on p. 12. Last 
year, multiple schools, many of them charter schools, failed to report the correct set of scores or report scores 
from different test administration for the same groups of children, resulting in errors on their profile on the 
state report cards that may inaccurately indicate students are performing or achieving at lower levels than 
they actually are. 

CHARTER SCHOOL RESPONSIBILITIES REGARDING TESTING 

Despite the fact that most charter schools do not have the financial resources to hire a person whose sole 
responsibility is that of test coordinator, charter schools need to appoint someone on their staff whose 
responsibilities include: 

1) keeping current with testing changes and responsibilities, 

2) receiving training on current testing practices and policies, 

3) educating staff at their school on these matters, 

4) coordinating testing so that all students are tested and scores are tracked over time, and 

5) reporting testing results to the state. 

This person could be the principal or his or her designee, but having a central person whose responsibility is to 
learn everything about testing that they can is increasingly important for several reasons: public accountability 
is almost entirely based upon testing outcomes; federal requirements have resulted in increasingly complex 
testing protocols; and that student achievement and promotion are now almost solely tied to test scores and all 
educators must do everything they can in order to help students succeed. 

HOW CHARTER SCHOOLS SHOULD COLLECT AND REPORT TESTING DATA 

Last year s situation in which charter school failed to compute test scores correctly underlines the importance of 
having a systematic plan whereby testing procedures are reviewed annually and one person is responsible 
for being the school s testing guru . Last year, multiple charter schools in Ohio failed to compute test scores 
correctly and request changes be made to their local report cards; although they had 15 opportunities to 
make such requests to the State department of Education. As reported in the Dayton Daily News, errors 
resulted from schools inability to track students scores over the course of the year and thus they misreported 
student scores. In addition, charter school principals indicated they did not know they could make changes 
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regarding which test scores were reported to the state, despite multiple attempts by the Ohio Department of 
Education to publicize this policy. Having someone at the school-level coordinate a systematic testing plan and 
assuming responsibility for educating themselves on Ohio testing procedures could rectify both situations. 

Thus all schools need to develop a systematic plan to ensure that: 

1) all students are tested using the appropriate test, 

2) makeup tests are given to transfer students or students who were absent on testing dates, 

3) multiple test scores can be linked to the same student, 

4) appropriate test results are reported to the state, and 

5) state test results are reviewed for accuracy. 

Again, despite the fact that charter schools may not be able to hire someone whose sole responsibility is that 
of a test coordinator, someone at the school must be responsible for ensuring a systematic plan is in place and 
reviewed annually. 

FINAL NOTES: 

Based upon new state and federal testing rules, the responsibilities that charter schools must bear as part of 
the testing requirements are great. However, Compass Consulting Group could provide technical support to 
schools to help them use their test scores to guide their instructional practices. Possible areas of technical 
support include: 

1) Further analysis of student-level data to assess growth and gains aggregated by school and grade 
and disaggregated by demographics such as IEP, LEP, FRL status, gender, ethnicity, etc. 

2) A 1-day workshop whereby CCG could work directly with school leaders and teachers in providing 
technical assistance to help them better understand 1) how the data was generated, 2) how the data 
can be interpreted, and 3) how implications regarding data results are formulated. 

3) An outline of technical assistance schools should seek to inform instructional practices based upon state 
data results. Such an outline would explain to schools 1) the possible implications of findings an dhow 
it relates to their instructional practices and policies and 2) implications for teacher professional 
development. 
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Appendices 

Ohio Test Development Process 



Overview 

The Ohio Department of Education facilitates the development of Ohio s assessments. Test development is a 
collaborative process between Ohio educators, assessment experts, education stakeholders, and testing 
companies. Hundreds of Ohio educators serve on development committees that represent the state 
geographically, ethnically, by gender, and by school district characteristics. The involvement of these 
participants and the Departments rigorous review procedures ensure that our assessments are valid and 
reliable. In other words, these contributors ensure that the assessments are appropriate in terms of content and 
difficulty and are an accurate and consistent means of measuring student mastery of the academic content 
standards. 

Development Steps 

1. Identification of Purpose 

Senate Bill 1 (signed by Governor Taft in June 2001) mandates that Ohio s assessment system be comprised 
of diagnostic, achievement, and graduation assessments all of which must be standards-based (or criterion- 
referenced). Each instrument plays a particular role in the overall assessment framework and therefore must 
be designed and developed to fulfill its function. The Ohio Department of Education consulted testing and 
instructional experts, as well as Ohio educators, in defining the precise purpose of the assessments and 
developing appropriate methods for developing and administering each. 

2. Test Blueprint Development 

After the assessments and their purposes have been defined, Content Advisory Committees comprised of Ohio 
educators continue the test development process by working with testing companies in formulating test 
blueprints (or test specifications). The committees review the academic content standards to determine the 
proportion of test content that ought to be devoted to each standard in order to provide the best and most 
balanced coverage. The committees consult the testing company and other testing experts to ensure that the 
test blueprints call for a test length, number of items and item formats that, according to research, yield the 
most valid and reliable measure of students mastery of the academic content standards. For example, the 
testing experts may give advice as to whether a constructed-response item is the best method for measuring a 
particular benchmark or grade-level indicator. 

3. Item Writing and Review 

The testing company develops and delivers items to the Ohio Department of Education. Department curriculum 
experts conduct a preliminary review of the items to ascertain content accuracy and alignment with the 
academic content standards, as well verify that the items follow established stylistic and grammatical 
guidelines. Department assessment staff coordinate the item review processes and ensure the standardization 
of item document formats. 

After this preliminary review, Ohio educators and other stakeholders thoroughly review the items. A 
Fairness/Sensitivity Committee, composed of Ohio stakeholders of diverse backgrounds, carefully reviews 
each item to ensure that it does not unfairly disadvantage particular groups of students. Then, a Content 
Advisory Committee, composed of active Ohio educators and other content experts, reviews each item to 
ensure that it is grade-level appropriate and measures the content it is intended to measure. (Five Step Item 
Review Process) 

4. Item Field Testing 

All items are field tested prior to their operational use. In other words, items are tested on actual students to 
make sure that they are satisfactory before they are used on a test that counts. The purpose of field-testing 
items is to get student data that will help in gauging item difficulty and identifying troublesome items. For 
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existing tests, new items are field tested on the operational form of the test. In other words, field-test items 
are embedded in the operational test but do not count toward a students score. Items field tested for a test 
that has yet to be administered are done so on an independent field-test form. 

5. Data Analysis and Review 

The Content Advisory Committee and Fairness/Sensitivity Committee once again review the items, this time 
equipped with data gathered from the field-testing. The committees consider the performance of certain 
student subgroups on each item when determining whether the items are fair and measure appropriate 
content. These last reviews from the committees complete the five-step item review process. 

6. Test Construction 

The test blueprint is refined based on information gathered during field-testing. The type (such as multiple 
choice and constructed response) and number of items may be modified if, for example, field-testing indicates 
that students had too much or too little time to complete the test. Once the blueprint has been finalized, the 
items reviewed and approved by the Ohio educator and stakeholder committees are used to construct a test. 
The test is then administered the following school year and includes embedded field test items which will be 
used to construct future tests thereby recommencing the development process beginning with item writing 
and review. 

Content Advisory Committees 

These committees are specific to a given test (e.g., Third Grade Reading Achievement Test, Ohio Graduation 
Test in mathematics). The purpose of these committees are to review and evaluate test questions and related 
test materials to ensure that each question is a valid and appropriate measure of the academic content 
standards for that particular subject area and grade-level. 

• Fairness/Sensitivity Committees 

These committees are associated with all tests related to a grade level or combination of grade levels (e.g., 
all five Ohio Graduation Tests, all kindergarten - third grade tests). The purpose of these committees is to 
review and evaluate test questions and related test materials to ensure that test questions are fair and 
unbiased for all groups of Ohio students. 

• Rangefinder Committees 

These committees are specific to a given test (e.g., Third Grade Reading Achievement Test, Ohio Graduation 
Test in mathematics). These committees are necessary whenever a test contains 'open-ended' questions (i.e. 
non-multiple choice questions). These committees establish the range, from no credit, to partial credit, to 
complete credit, of acceptable student responses. 

• Standard Setting Committees 

These committees are specific to a given test (e.g., Third Grade Reading Achievement Test, Ohio Graduation 
Test in mathematics). Unlike other advisory committees, these committees meet only once, to recommend the 
performance standards (i.e., the score for basic, proficient, and advanced achievement) for a given test. 
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Testing Dates 2003-2004 



Third Grade Reading Achievement Test 

October 6-10, 2003 Fall Administration 

(Five days including make-up testing) Results Date - To Be Determined 
March 8-1 2, 2004 Spring Administration 

(Five days including make-up testing) Results Date - To Be Determined 
July 5-26, 2004 

(Summer Administration) Results Date - To Be Determined 

NOTE: If a student attains a score of Limited on the Third-Grade Reading Achievement Test, the 
district is obligated to do one of the following: 

• Promote the student to fourth grade if the students teacher and principal agree that other 
work done by the student demonstrates satisfactory reading skills and readiness to be 
promoted to the fourth grade; 

• Promote the student to fourth grade, but provide the student with intensive intervention services 
in fourth grade; or 

• Retain the student in third grade. 

Fourth Grade Proficiency Test 

October 6-10, 2003 Fall Administration 

(Reading only - includes make-up testing) ^Results Date - November 21, 2003 
March 8-26, 2004 Spring Administration 

(Every other day administration of all five tests plus nine calendar days of make-up testing) *Results 

Date - May 25, 2004 

July 5-26, 2004 Summer Administration 

(Reading only - includes make-up testing) ^Results Date - No later than August 26, 2004 
individual student results sent to districts/schools. 

NOTE: This school year will mark the last fall, spring and summer administrations of the Fourth-Grade 
Reading Proficiency Test, 



Sixth Grade Proficiency Test 

March 8-26, 2004 Spring Administration 

(Every other day administration of all five tests plus nine calendar days of make-up testing) *Results 
Date - May 25, 2004 

individual student results sent to districts/schools. 

Ninth Grade Proficiency Test 

NOTE: Ninth grade students do not take the Ninth Grade Proficiency Test this year eligible 
participants are 1 0 ,h , 1 1 th , and 1 2th grade students who have not previously passed one or more of 
the tests or 1 0 th , 1 1 th , and 1 2th grade students who have not taken the tests. 

October 27-November 9, 2003 Fall Administration (Grades 10-12) *Results Date - Senior results by 
December 4, 2003. All other results by January 8, 2004. 

March 1-14, 2004 Spring Administration 

(Grades 10-1 2) ^Results Date - Senior results by April 1 8, 2004. All other results by May 1 3, 2004. 
May 1-10, 2004 Special Administration 

(For seniors only) Results Date - Senior results by May 24, 2004 

July 5-26, 2004 Summer Administration Results Date - Results no later than August 30, 2004 
individual student results sent to districts/schools. 
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Ohio Graduation Test (Reading & Mathematics) 

March 15-19, 2004 Early Spring Administration 

(ALL 10th graders take reading and mathematics; 5 days to include make-up testing) Results Date - 
Be Determined 
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Proficiency & Testing Contacts and Information 

http;//www.ode.state.oh.us/proficiency/contacts.asp 

Phone: 614-466-0223 
FAX: 614-728-7434 

Jan Crandell, Director 

Sharon Bowman, Item/Test Development 

Joanna Cruse, Secretary 

Paula Mahaley, 4/6/9 Team Leader 

Jack Schwarz, Scoring/Reporting/Oral Administration 

Debbie Metzger, Administrative Assistant 

Shelly Thrash, Management Analyst 

Glenda Zempter, Consultant 4/6 Scoring/Reporting 

Max Xu, Associate Director 

Sue Rapier, NAEP Administrator 
Kurt Taube, Scoring/Reporting 
Linda Bartman, Secretary 

Nancy Haefeli, Associate Director 

Barbara Sisco, Consultant 
Tom Bulgrin, Consultant 
Barry Lowry, Consultant 
Donald Peasley, Consultant 
Diana Viers, Administrative Assistant 

Specific Testing Area Information 
Area, Contact Name, and E-mail Address 
4-6-9 Tests Paula Mahaley 
Ohio Graduation Test Tom Bulgrin 
Alternate Assessment Don Peasley 
K-5 Barry Lowry 

LRC and Accountability Jill Dannemi 
AYP Max Xu 

Test Contractor Contacts 
4th and 6th grade tests 

Larry Ehret, NCS Pearson 
Larry.Ehret@pearson.com 

9th grade tests and OGT 

Teri Broderick, Questar 
tbroderick@questared.com 

K 5 diagnostic and achievement tests 

June Zack, AIR 
jzack@air.org 

PROFESSIONAL DEVELOPMENT OPPORTUNITIES BOOKS, ARTICLES, WORKSHOPS, ETC. 



Paula.Mahaley@ode.state.oh.us 

Tom.Bulgrin@ode.state.oh.us 

Don.Peasley@ode.state.oh.us 

Barry.Lowryl@ode.state.oh.us, 

Jill.Dannemiller@ode.state.oh.us 

Max.Xu@ode.state.oh.us 
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• Hutton, David. W. From Baldrige to the Bottom Line: A Road Map for Organizational Change and 
Improvement. American Society for Quality: Milwaukee. 20001. ISBN: 0-87389-473-1 

• Beecroft, G. Dennis et al., Editors. The Executive Guide to Improvement and Change. American Society for 
Quality: Milwaukee. 2003. ISBN 0873895797 

ODE WEBSITE LINKS 

• The Ideas of ODE: Monthly Information from the Offices of Curriculum, Instruction, and Assessment 
http://www.ode.state.oh.us/proficiencv/Monthly communications.asp 

• Ohio State Department of Education Testing Rules Book 
http:/ /www.ode.state.oh.us /proficiency /PDF /RulesBookSept02. pdf 

• Test Coordinator's Listserv 

http://lyris.ode.state.oh.us/cq?-bin/lyris.pl?enter=test-adm-coor 

Curriculum Coordinator's Listserv 

• http://lyris.ode.state.oh.us/ cqi-bin /lyr is. pl?enter— curriculum coord 
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Glossary of Ohio State Testing Terms 

Academic Content Standards 

The standards that delineate what a student should know and be able to do at each grade-level. The 
academic content standards are composed of standards, benchmarks, and grade-level indicators. Ohio s 
statewide assessments are based on the academic content standards. 

Achievement Tests 

Like the diagnostic assessments, achievement assessments also must measure student mastery of academic 
content and related skills but are designed to serve additional purposes. At the student level, certain 
achievement assessments (e.g., Ohio Graduation Tests) figure into the criteria for obtaining a diploma. State 
law also mandates that school districts provide intervention services for students who fail to obtain designated 
achievement levels. 

On a broader level, achievement assessment results will inform decisions for educational programming at the 
school, district and state levels. In order to further the usefulness of results for this purpose, the ODE must 
disaggregate statistically sound assessment results according to student age, race and ethnicity, gender, 
length of enrollment in a particular district or school, economic background and any other category the ODE 
deems appropriate. 

In addition to their value for improving educational programming at the school, district and state levels, the 
achievement assessment results will be used for state and federal accountability purposes. As part of the 
Adequate Yearly Progress (AYP) calculations, ODE will report to the federal government data from 
achievement assessments in reading and mathematics and eventually science. At the state level, results of all 
achievement assessments factor into the performance indicators used for the school and district report cards. 

Due to their uses for accountability purposes, it is critical that these assessments yield valid and reliable results 
for students across the state. Special measures are taken to assure that the assessments and their 
administration yield such results. For example, each test must be scored in the same way and test results must 
be carefully reviewed for accuracy. Specially trained personnel score all achievement tests in order to ensure 
this accuracy and consistency. 

Benchmark 

A specific statement in the academic content standards of what a student should know and be able to do at a 
specific time in his/her schooling. Benchmarks are used to measure a student s progress towards meeting a 
particular standard. Benchmarks are defined at the end of specific grade bands for each content area. For 
example, there are mathematics benchmarks for each of the following grade bands: K-2, 3-4, 5-7, 8-1 0 and 
1 1-12. Virtually all of Ohio s achievement assessments will be based on the benchmarks in the academic 
content standards. 

Constructed-Response Item 

An item that requires the student to construct a response, instead of selecting the answer from a set of 
choices as with a multiple-choice item. These types of items have a greater point value than a multiple- 
choice item, typically take more time to answer, and must be scored by skilled scorers. Ohio uses two types of 
constructed-response items: short answer and extended response. 

Confenf Advisory Committee 

The Content Advisory Committee is comprised of active Ohio teachers, parents of children in school (K-l 2), 
school board members, representatives of educational organizations (e.g., OEA, OFT, as well as content 
organizations like OCTM), and business/community members. Diversity within these groups is reflective of 
Ohio s diverse populations and regions. 

In addition, Ohio educators represent the diversity of Ohio s schools and districts: public, nonpublic, joint 
vocational, urban, suburban, and rural. Some educators represent higher education or administration, but the 
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majority of the members on each committee are classroom teachers with a variety of years of teaching 
experience in the appropriate grade level and content area. 

The role of the Content Advisory Committee is to make recommendations to the Ohio Department of Education 
(ODE) to ensure all assessment items: 

• are aligned with the academic content standards; 

• contain appropriate grade level content; 

• have the appropriate level of difficulty for the grade level; 

• have one and only one correct answer (multiple-choice items only); 

• have scoring guidelines which reflect the intended student response (constructed-response items 
only); and 

• are clearly written. 

Criterion-referenced Assessment 

An assessment in which a student's performance is compared to a specific learning objective or achievement 
standard (e.g., academic content standards) and not the performance of other students. Criterion-referenced 
assessments measure how well students are achieving on specific goals or standards rather than measuring 
how well their performance compares to a norm group of students nationally or locally (as with norm- 
referenced assessments). In criterion-referenced assessments, it is possible that all of the students tested will 
reach a particular goal or achievement standard. 

Diagnostic Assessments 

The diagnostic assessments will serve as district-based tools for monitoring progress at grade levels in which 
an achievement assessment is not administered. The purpose of the diagnostic assessments is to inform 
instruction; they are not for state accountability purposes. The assessments will be based on the grade-level 
indicators and therefore will serve as a monitoring mechanism to ensure that students are progressing toward 
the achievement of the academic content standards and success on subsequent achievement tests. Districts will 
handle the scoring and reporting for diagnostic assessments; scores will not be reported to the state. 

Ohio law requires school districts to administer diagnostic assessments at least once annually and within 30 
days of a student transferring into a school district or school. Each district is required to provide intervention 
services to students whose diagnostic assessments show that they are failing to make satisfactory progress 
toward attaining the academic standards. 

The diagnostic assessments will consist of a screening measure (a quick analysis at the beginning of the year 
or upon a new student s arrival), a diagnostic measure (to monitor student progress and gather information to 
inform instruction), and an observation measure (to inform and improve instruction by systematically collecting 
evidence of students knowledge and skills). 

Fairness /Sensitivity Committee 

The Fairness/Sensitivity Committee is comprised of Ohio citizens who are selected to represent the racial, 
ethnic, gender, and cultural diversity that exists in the State of Ohio. Committee membership reflects areas of 
interest or experience such as bilingual/multicultural education, ethnic and religious studies, and children with 
special needs. 

According to Ohio Revised Code, Section 3301.079(G), the role of the Fairness/Sensitivity Committee is to 
prevent any question on any state assessment to include, be written to promote, or inquire as to individual 
moral or social values or beliefs. The decision of the committee is final regarding this matter. 

Other roles of the Fairness/Sensitivity Committee include ensuring that items do not disadvantage groups of 
students because of their race, ethnicity, gender or disability, as well as ensuring that diverse cultures are 
represented in assessments and that material used is neither offensive nor stereotypes any student group. 
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Field-Test Items 

Items that are administered to students but do not impact their scores. Field-testing is an integral aspect of the 
item development process that enables the state to test an item and review its performance. All items 
successfully complete the five-step item review process prior to their operational use, so no student is scored 
on an item that has not been field-tested. 

Five Step Item Review Process 

1 . The Fairness/Sensitivity Committee looks at each item to ensure that no group of students will be 
disadvantaged. The committee may choose to change the wording of an item or discard it altogether. 

2. The Content Advisory Committee (one for each content area) reviews each item to ensure that the 
content being assessed is grade-level appropriate. As with the Fairness/Sensitivity Committee, the 
Content Advisory Committee may change the wording or discard an item. 

3. Items are field-tested in circumstances similar to those for operational testing. Items will also be 
embedded on operational test forms to continually add potential items to the item bank 

4. The Fairness/Sensitivity Committee again reviews field-tested items with information regarding the 
performance of identified sub-groups on each item. Items that require modification must be field 
tested again. 

5. Field-tested items are then reviewed by the Content Advisory Committee once again with the sub- 
group data from the field test. As with the Fairness/Sensitivity Committee, items that require 
modification must be field tested once more. 

Grade-level Indicator 

A specific statement of the knowledge and/or skills that a student must demonstrate at each grade level. 
Grade-level indicators serve as checkpoints that monitor progress toward the benchmarks of the academic 
content standards. The diagnostic assessments will be based on grade-level indicators. 

Item 

A test item is a question, prompt or task that requires a students response. An item may include supporting 
text or graphics and may be in the form of multiple-choice or constructed-response. 

Item Bank 

A database that contains items that are eligible to be used on operational tests. Only items that successfully 
complete the 5-step review process are placed in the item bank. 

Multiple-Choice Item 

An item that involves a students choosing from a given set of possible answers. 

Norm-referenced Assessment 

An assessment that compares student performance with that of a larger group (the norm group). Usually the 
larger group or "norm group" is a national sample representing a wide and diverse cross-section of students. 
Norm-referenced assessments contrast criterion-referenced assessment, which measure students against a set 
of standards or objectives. 

OWo Graduation Tests (OGT) 

The Ohio Graduation Tests (OGT) are achievement assessments administered in the 1 Oth grade. As with all 
achievement assessments, the OGT serve programmatic as well as accountability functions. In addition, all 
students will be required to pass all portions of the OGT in order to obtain a diploma. These tests will replace 
the 9th grade proficiency tests as a graduation requirement beginning with the class of 2007. There 
eventually will be Ohio Graduation Tests in reading, mathematics, science, social studies, and writing. 

Operational-Test Items 

The test items administered to students upon which student scores are based. All operational items must have 
successfully completed the 5-step item review process. 
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Public Release 

All operational test items given during a school year are made available for public review and reproduction 
each year. Previously administered operational tests are made available in either electronic or paper 
formats. Only when items are used to obtain a student s score can they be released to the public. 

Reliability 

Reliability is the degree to which the results of an assessment are dependable and consistently measure 
student knowledge and skills. Reliability is an indication of the consistency of scores across different tasks or 
items that measure the same thing, across raters, and over time. Thus, reliability may be expressed as the 
relationship between test items intended to measure the same learning or knowledge (item reliability), the 
relationship between two administrations of the same test to the same student or students (test/retest 
reliability), or the degree of agreement across two graders (rater reliability). 

Standard 

An overarching goal or theme. The standard statement describes, in broad terms, what students should know 
and be able to do as a result of the K-1 2 program. Mathematics, for example, has a total of six standards. 

Testing Companies 

Testing companies (or test developers) are companies with whom the Ohio Department of Education works to 
develop and administer Ohio s assessments. The Department awards contracts on a competitive basis. 
Companies submit a proposal in response to an Invitation to Bid (ITB) or Request for Proposal (RFP). The 
Department then enters into a contract with the company that presents the best and most cost-effective 
proposal. 

Validity 

Validity is the extent to which an assessment measures what it is supposed to measure and the extent to which 
inferences and actions made on the basis of test scores are accurate and appropriate. For example, if a 
student performs well on a reading test, how confident are we that the student is a good reader? A valid 
standards-based assessment is aligned with the intended learning and knowledge to be measured and 
provides an accurate and reliable measurement of student achievement relative to the standard. 



BEST COPY AVAILABLE 



r 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 

Reproduction Basis 




This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 



This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



c FF-089 (1/2003) 




